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Earlier work inspired by a body of literature raised important questions about the workings of the UK’s Research Excellence Framework 
(REF) and its predecessor the Research Assessment Framework (RAE), and noted the possible adverse outcomes of such processes. This 
paper builds on this by examining the findings of a small survey of social science academics. The survey identified concerns about the 
validity of the REF as a proxy for quality, and the role it has had in shaping patterns of research behaviour. There were also frequent 
concerns related to morale. Yet although responses tended to be negative, there was also a significant voice signalling the importance the 
REF plays in ensuring accountability and transparency in research, as well as a sense that the pressures that come with such processes 
are simply ‘part and parcel’ of academic life. The role of wider time-management factors, related to heavy teaching and administration 
burdens, was also signalled, and cited by some as overshadowing the pressures of REE 
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Introduction 

The UK’s Research Excellence Framework (REF) is used 
to determine the quality’ of research activity within 
UK universities, which subsequently informs research 
funding allocation from the major funding bodies (REF, 
2012). Replacing the ResearchAssessment Exercise (RAE), 
the REF has just completed its first cycle involving the 
assessment of research undertaken between 2008 and 
2013.This utilised 36 units (subject areas) of assessment, 
each assessed by sub-panels of subject experts. The 
submissions from universities were placed in categories 


of overall quality, from one-star’ (1*) to four-star’ (4*) (as 
well as an unclassified’ category). These categories were 
determined by the weighted sub-profiles of ‘output’, 
impact’, and environment’. Research rated as ‘4*’ 
indicates that it is world-leading’, whilst one-star’ denotes 
research of national recognition (REF, 2012). The results 
were published in December 2014 and have prompted 
a multitude of press reporting and claims of success 
from universities across the UK, who arguably chose to 
interpret the results in the manner most sympathetic 
to them. On the REF results day, The Times Higher 
Education published a table of excellence’ pertaining to 
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the overall grade-point average for university submissions 
(Jump, 20l4a).The same article also offered an alternative 
ranking, this time based on grade-point average 
specifically relating to impact’. This offered a revised 
order of universities. On the same day, the Telegraph 
published a different ranking, this time based on research 
power’, where the actual volume of research is integral 
to the placing of an individual institution (Telegraph, 
2014).This ranking is a closer proxy to the likely funding 
allocations as a result of the REF and, importantly, varies 
significantly from other rankings. The proportion of 
4* research, or proportion of 4* plus 3* have also been 
used to confirm ranking. Not surprisingly, on university 
websites and in press releases, universities carefully chose 
which measure to cite. Thus, any one of a handful of 
universities can claim to be the best’ in the country for 
research. The Guardian picked up on this in an article 
that asked whether the REF has been drowned out by its 
own noise’. In the Guardian article, it was suggested that 
when 25 university departments can claim to be in the 
top three for research in their field, which appears to be 
the case amongst the REF 2014 fallout, then the value of 
the REF comes in to question (Wolff, 2015). Seemingly, the 
publication of the REF results and their malleability has 
caused some confusion and controversy, often in a similar 
vein to the actual processes of the exercise itself. 

Yet, as some have noted, research assessment exercises 
such as the REF have important implications and are 
linked to a number of matters such as accountability 
and efficiency of research (Ab Iorwerth, 2005). Equally, 
a number of papers have reported on the possibility 
for adverse outcomes from such assessments (e.g. Hare, 
2003; Bowring, 2008; Wells, 2013 and so on). A number 
of authors have signalled their dissatisfaction with the 
workings of the REF, pointing to its potentially divisive 
and morale-sapping nature (e.g., see Jump, 2014b), and 
the potential for it to adversely shape the nature of 
research being conducted. Townsend (2012), like other 
authors, has highlighted the danger that the REF might 
restrict the type of work being done to meet the criteria 
of inclusion in the REF. Wells (2013) also explored this 
possibility. The actual effort involved in meeting the 
submission requirements, for example in preparing 
‘impact case studies’, has also been questioned by some 
(e.g., see Jump, 2014b).The requirement to demonstrate 
impact’ and the weighting placed on this within the 
assessment was particularly noteworthy in REF 20l4.This 
has been viewed as problematic for some time (e.g., see 
Miller & Sabapathy, 2011). Watermeyer (2012, 2014) has 
detailed the context within which ‘impact’ has figured 


so prominently in this cycle, and equally, the resistance 
this has generated, where impact’ has been understood 
by academics: 

...as an infringement to a scholarly way of life; as 
symptomatic of the marketisation of higher education, 
and as fundamentally incompatible and deleterious to 
the production of new knowledge (Watermeyer, 2014, 
p. 1). 

Following the fall-out from the publication of results 
for REF 2014, Jump (2015) explored the role of impact', 
particularly impact case studies’, and the possibility for 
game-playing to have occurred at the level of institutions, 
as well as REF panels. In the case of the latter, this was 
suggested as a possible ploy by panel members to 
ensure their discipline was not perceived to have been 
underperforming in relation to ‘impact’ and found wanting 
in comparison with other disciplines. 

In a similar vein to much of the above commentary, 
Murphy and Sage (2014) found that academics reporting 
on the REF tended to be sceptical about it in one form 
or another. Often, this was related to a discussion about 
impact’, and in a related sense, the demands of proving 
impact’ (see HEFCE, 2010 for a discussion of impact’), 
or wider anxieties created by the demand to prove 
your worth (Murphy & Sage, 2014). The paper also 
demonstrated that, although reporting on the REF was 
primarily (but not exclusively) negative, the level of this 
varied according to author type and their institutional 
base. It was also shown that those author characteristics 
appeared to shape the types of concerns being raised 
in relation to the REF. Yet, themes such as impact’, 
funding , and marketisation’were most prominent, all of 
which appear to be connected. Authors also seemed to 
be concerned with how the REF might adversely shape 
researcher behaviour; narrowing the type of research 
undertaken. This concern is reflected elsewhere (e.g. 
Watermeyer, 2012, 2014). There are also concerns that 
the pressures of playing the game can adversely shape 
behaviours, possibly creating incentives to cut-corners 
- the possible outcomes of this have been explored 
previously (e.g., see Fanelli, 2010; Murphy, 2013). 

The REF, like the previous guise of the RAE, has high 
stakes: and universities have invested heavily in the 
process, including the buying-up of researchers to boost 
REF scores (Jump, 2014b). Unsurprisingly controversy has 
inevitably followed. Academic researchers and managers 
alike have had to dance to the tune’ of the REF despite 
genuine concerns about the nature of the processes 
involved and possible adverse outcomes. Seemingly, 
the implications of this extend beyond the UK. Such 
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research assessment exercises exist in many countries: 
the Performance-based Research Fund in New Zealand; 
the Netherlands Research Embedment and Performance 
Profile; and the Australia Research Quality Framework 
(now Excellence in Research for Australia - ERA) are just 
some examples (see Curtis, 2015). With this, the extent to 
which a balance is struck between ensuring excellence of 
research and value for money, with fostering morale and 
fit-for-purpose proxies for quality’, appears to vary. 

Our aim in this paper is to examine input from 
individual academics who responded to our survey, in 
order to determine the extent to which those voices 
reflect or challenge the issues raised in our earlier paper 
and in other recent commentary on the REF. 

Methodology 

We disseminated an online questionnaire to academics 
in England and Scotland in late 2012.As social scientists 
we focused on that section of academia. The survey link 
was sent to those identified as gatekeepers to academic 
mailing-lists within social science departments and 
groupings within institutions; this varied according 
to institution, but tended to be senior administrators, 
departmental heads and subject leads.We split institutions 
into their associated mission groups in order ensure that 
we targeted a variety of university types: both research¬ 
intensive and teaching-intensive institutions. The 
questionnaire offered respondents a series of questions, 
each designed to assess the extent to which they viewed 
the forthcoming REF as a positive or negative process, 
what they associated with the process in terms of likely 
outcomes and any possible issues associated with the REE 
The response rate to our questionnaire was relatively low, 
despite us sending two waves of requests. We received 
64 completed surveys from the pool of 33 institutions 
we contacted. The responses did however raise some 
interesting questions. 

The final question in the survey was an open-ended 
request for any other comments’.We found this to be of 
particular interest owing to the level of qualitative data 
obtained, thus our main emphasis within this paper is 
to analyse these responses. We received 32 qualitative 
statements, some of which were particularly detailed 
- perhaps suggesting that some academics had a lot to 
say about the REF. The responses came from a mixture 
of both research and teaching-intensive institutions and 
from both junior and senior academics. The data can by 
no-means be viewed as representative of academia as a 
whole but, when viewed more widely in the context of 


other work in this area, highlight some important issues 
in need of attention. 

Results 

Respondents to the survey were more or less equally split 
between men and women, and they affiliated themselves 
with a range of social science subjects, across both 
research-intensive and teaching-intensive institutions and 
both qualitative and quantitative methodologies. Most 
were either senior lecturers or lecturers, and just over 
half had participated in a previous RAE. On reviewing 
the qualitative data, we identified five themes of interest. 
Some of the themes demonstrate some level of concern 
amongst academics about the way in which the REF has 
worked and its anticipated outcomes. However, there was 
also some level of acceptance, and even positivity about 
the REF, and some academics pointed to the wider context 
of academia as being of more relevance than the REF to 
the possible pressures and stresses that academics might 
face.The emergent themes from the qualitative responses 
are presented below and, where appropriate, they are 
supplemented with data from the structured elements of 
the survey. 

‘REF-able’ work only 

A number of respondents were concerned that REF 
narrowed the type of research being conducted and the 
type of publications encouraged within departments, where 
only certainly types of work and publications were deemed 
to be REF-able’. For example, one noted that the REF: 

‘dictates what people write and research, tinder-val¬ 
ues theoretical work... and deters academics from 
embarking on major long-term projects’. 

In a related sense, another respondent reflected 
negative experiences associated with the prioritisation of 
research within departments: 

‘projects are being turned down, longer-term - and 
arguably more interesting and more internationally- 
relevant - projects are being turned down as ‘too 
ambitious for this REF’ and funding is being given to 
those who already have full, relatively strong, submis¬ 
sions'. 

For other researchers, a major issue was how the 
REF incentivised the production of measurable impact 
factors’, rather than research that academics considered 
to be ‘socially and politically important’. Indeed, another 
respondent noted how the measurement of research’ 
inherent within the REF was, in her eyes, ‘distasteful, 
difficult and against the principle of academic freedom’. 
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This was a common feeling amongst respondents: that the 
REF constrained and limited academic research. 

In addition, the following reflected concerns for how 
publishing habits were being shaped by the requirements 
of REF submissions: 

‘The more academics are pressured to publish their 
work in more elite journals, the less we are able to 
communicate with and disseminate our ideas to the 
public sphere’. 

Perhaps then there is the possibility that competing 
agendas exist, where true impact’ of work and the 
prestige of publication do not always mirror each other. 
Another respondent noted the problems associated with 
the specific situation of co-authored papers, where they 
count for the REF in some instances, such as external 
collaborations, but not for others, such as internal 
collaborations. Whilst another statement focused on the 
relatively low standing of books within the REF process: 

I have published a book with a top publisher in the 
current REF period, I have been deterred from pub¬ 
lishing another book because of the need to get arti¬ 
cles in top journals.’ 

A different respondent noted how: 

I was told by an external reviewer that this [human 
rights research] did not count for REF purposes as it 
is not purely scholarly work’ and not written in the 
appropriate ‘scholarly form’. If such work does not 
count for the REF, then there is something seriously 
wrong with the REF.’ 

In the body of the survey, pressures to publish were 
cited as influencing the nature of work being undertaken. 
Although the responses varied significantly across the 
scale we used, there was a slight skew towards responses 
at the higher levels when respondents were asked 
whether academic freedoms were compromised by the 
demands of the REF, and the pressures to publish. 

The REF as an ineffective measurement 

Many of the surveyed academics were concerned with the 
actual processes associated with the REF: how this might 
be subject to game-playing’ and how such processes 
missed the point in terms of effectively encouraging, 
assessing and rewarding research quality. For example, 
one respondent noted how despite acknowledging that 
accountability of research was important: 

‘the way in which the government does it through the 
REF (and QAA, the Quality Assurance Agency) is the 
worst possible way, since it imposes enormous costs 
of administration and seeking to game the system and 
measures quality very poorly. ’ 


Impact case-studies were cited within the responses 
as being time-consuming and, more widely, the costs and 
energies of the administrative demands of the REF were 
noted by some, for example: 

The time that I have already had to spend on the 
administrative side of the REF (particularly impact 
case-studies, but also reporting information through 
cumbersome online bureaucratic systems) is so enor¬ 
mous that I could have written, realistically speaking, 
at least three additional articles or half a book manu¬ 
script in the time that has been consumed.’ 

Another issue generated in the responses concerned 
the REF requirement to submit just four outputs, which 
was perceived to have meant that prolific researchers 
are not duly rewarded for their sustained and successful 
engagement with research. 

Where respondents were asked about their 
participation in REF 2014 within the structured part of the 
questionnaire, 35 of the 60 respondents to that question 
went on to participate, five did not have the necessary 
publications, three were to opt-out, and interestingly, 
the rest were unsure. Thus seemingly, the REF prompted 
some confusion amongst academics about what might 
qualify them for inclusion or, alternatively, whether their 
department intended to use their outputs. 

One respondent saw the inherent positives of such 
a measurement instrument, but argued that processes 
associated with the REF essentially offers management a 
tool to apply undue pressure on its staff; this is reflected 
more broadly in our later discussion about morale. 
Several respondents additionally mentioned the very 
real prospect of researchers and institutions gaming the 
system’, to the detriment of the actual aims of the REF 
vis-a-vis encouraging and rewarding quality of research. 
Another response suggested that: 

'the REF has produced greater attempts at managerial/ 
top-down influence on research direction. It skews the 
balance between teaching and research, effectively 
‘dumbing down’ both’. 

Concerns for morale and careers 

A concern for how the REF and the processes associated 
with it adversely impacted on staff morale appeared often 
within the qualitative responses and also in the more 
structured element of the questionnaire. For example, one 
qualitative statement noted how the pressure to publish 
together with increased difficulty in getting published has 
an impact on morale’, whilst another stated, in relation 
to processes such as the REF, that it feels as though ‘it 
undermines much of our work'. 
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Another respondent noted how the REF determined 
that only research-prolific staff were considered‘attractive’ 
to prospective employers, who may simply be looking to 
hire REF-able' academics. Yet periods of maternity leave 
had meant that her publication record was not what it 
would have been' and, consequently, she claimed that the 
wellbeing of my family is impacted to an extent in that I 
cannot get work closer to home...It all comes down to 
publications’. 

Another explained how they sought to leave UK 
academia to work overseas.The structured element of the 
survey supported some of these accounts. For example, on 
being asked whether the pressures to publish had made 
them consider changing jobs as a result, 23 out of the 60 
respondents for that question positioned themselves in 
the top four scores on our scale - indicating that they had 
considered their position with some intent. 

Self-imposed pressures 

A number of respondents noted that they had not 
experienced pressures to publish, either because they 
were new in their role, had been well-supported by 
their department, or they did not define themselves as 
a researcher. Others were fairly staunch in noting how 
pressures to publish in academia, whether related to the 
REF or otherwise, were part and parcel’ of working in 
the sector and often self-imposed: even if some did not 
agree with the way in which the REF was conducted. For 
example: 

‘The pressure I feel to publish is as much about me 

seeing publication as a way to career development 

beyond my current institution.’ 

Similar opinions were offered by other respondents 
who, despite the pressures to publish inherent in the REF 
process, stated that a proportion of this pressure came 
from them, with publishing being seen as fundamental to 
an academic career and research dissemination. Further, 
some respondents claimed that institutions had increased 
the amount of support given to staff to concentrate 
on publishing research as a consequence of the REF. 
Nevertheless, a common feeling was that although 
publishing was an integral part of academic work, the REF 
process could sometimes act in a corrupting way, with 
one respondent noting that although publishing was the 
‘best part of my job’ and a main motivation', the overall 
measurement strategy was distasteful’. Thus, seemingly, 
feelings towards the relationship between the REF 
process and the publishing process are more balanced 
than earlier themes might have signalled. 


Time management is the issue 

On being asked about pressures to publish on a scale of 
1 to 10'within the structured element of our survey, the 
majority of the sample reported significant pressures, with 
46 of the 62 respondents for that question positioning 
themselves in the top four scores - the highest levels of 
pressure. Follow-up questions then asked whether such 
pressures had an impact on other academic duties, and 
a skew towards the higher levels was evident, but this 
was not as apparent as it was in the earlier question. A 
skew towards the higher levels was then demonstrated 
when respondents were asked whether there had been 
an increase over time in publishing pressures; 35 of the 
57 respondents here occupied the top four levels. Further, 
on being asked how those pressures might have affected 
them, respondents cited particularly longer working 
hours, change in expectations from management and less 
time for other academic duties (from a provided list). 

These feelings were also evident in the qualitative 
responses to the survey, with time management cited as 
one of the most important issues for academics. Amongst 
the qualitative statements, one respondent remarked that: 

‘The real problem is the increasing burden of teach¬ 
ing and administration.. .1 am now doing more teach¬ 
ing and administration (which I do not enjoy) than 
research. That is why I am considering a career 
change.’ 

In a related sense, one response indicated that much 
of what this questionnaire covers is really down to time 
management', whilst another stated that: 

‘my duties do not give me time to work on my publi¬ 
cations, I feel like I am cheating if I work on my own 
research when I should be doing admin in work time 
because the norm is that we’re expected to work on 
these outside of office hours.’ 

Such sentiment is reflected in this final remark: 

In my case and (I suspect) a number of others, pres¬ 
sures really originated through massive teaching and 
administrative burdens - leading to very high working 
hours and inadequate holidays, while still not being 
able to devote as much time as desired to research.’ 

Conclusions 

Although our survey elicited fewer responses than we had 
anticipated, the data did raise a number of concerns in 
relation to how the REF was viewed as having a negative 
impact on academia and the working environment within 
UK universities. Much of this corresponds with what we 
found in our 2014 paper and other recent reporting on 
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the REF (e g. Curtis, 2015;Jump, 2015;Watermeyer, 2014). 
Yet, this has to be put in to context. A number of the 
respondents noted that the REF has a valuable role to play 
in ensuring quality, value for money and allowing funding 
to follow excellence. This is perfectly understandable, 
provided the processes involved with the REF function 
in the manner that meets the intended outcomes without 
adversely shaping behaviours of staff and institutions. In 
this context, many of our respondents had bought in to the 
philosophy of such assessment instruments, even though 
some of them did signal scope for possible improvement. 

There was a significant voice pointing to wider 
industry pressures, such as heavy administrative and 
teaching burdens as occupying a more significant role in 
determining staff morale and pressures. It was also noted 
how pressures were often self-imposed, in the pursuit 
of promotion and self-development, and that this is part 
and parcel’ of working in academia. It must also be noted 
however that differences between institutional type and 
researcher seniority will most likely have influenced 
some of the interesting diversity of perspective towards 
the REF demonstrated in our data. 

The discussions around the REF have been more 
balanced than some would imagine, but they have 
still tended to be negatively skewed on the whole. Our 
analysis here suggests that many academics have genuine 
concerns about the implications of the REF affecting 
their morale, their sense of their role and, potentially, 
their employment within the sector. Yet some did adopt 
a more sympathetic view. As we and other authors are 
currently involved in examining the fall-out from REF 
2014, universities across the UK are readying themselves 
for the requirements of REF 2020 and the new challenges 
that this will provoke, which are set to include changes 
to the sub-profile weightings and an emphasis on open- 
access publications. It is hoped that lessons from the past 
are learnt. 

Tony Murphy, FHEA is a senior lecturer in criminology at 
Sheffield Hallam University, Sheffield, UK. 

Daniel Sage is a social policy researcher at the University of 
the West of Scotland, Paisley, UK. 
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